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[57] ABSTRACT 

A generalized compliant motion with sensor fusion 
primitive uses a set of input parameters provided from a 
local control site to a remote execution site to control a 
telerobot with a combination of a priori trajectory mo- 
tion and real and virtual local and remote sensor inputs. 
The set of input parameters specify the desired telero- 
bot behavior based on a combination of local and re- 
mote information. This general motion primitive re- 
quires less computer memory size, and provides more 
capabilities, than the task specific primitives it replaces 
because redundancies are eliminated while permuta- 
tions of capabilities are available. Trajectory motion 
occurs during a nominal motion time segment while 
termination conditions are monitored during an ending 
time segment to stop motion when a termination condi- 
tion occurs. Force and compliant motion, teleoperation, 
dither, virtual springs restoration and joint limit control 
are combined with the trajectory motion at the remote 
site. 

1 Claim, 3 Drawing Sheets 
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GENERALIZED COMPLIANT MOTION 
PRIMITIVE 

ORIGIN OF THE INVENTION 5 

The invention described herein was made in the per- 
formance of work under a NASA contract, and is sub- 
ject to the provisions of Public Law 96517 (35 USC 202) 
in which the Contractor has elected not to retain title. 
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CROSS REFERENCE TO RELATED 
APPLICATION 

This is a continuation-in-part of U.S. patent applica- 
tion Ser. No. 07/699,299, filed May 9, 1991, U.S. Pat. 
No. 5,231,693. 15 


TECHNICAL FIELD 

The present invention relates to robotic control sys- 
tems and, in particular, to task primitives, that is, pre- 
programmed programs for controlling a robot to ac- 20 
complish a task based on a set of input parameters 
which may be designated before, at the start time of, or 
during task execution. 

BACKGROUND OF THE INVENTION 2 5 

Techniques for the operation and control of robotic 
manipulators in some environments, such as manufac- 
turing facilities, have been developed and used success- 
fully for many years. Many such techniques include the 
creation of new computer programs for each new ro- 30 
botic task. 

These techniques must be substantially modified for 
use in poorly modeled environments, or to perform 
tasks in response to unplanned scenarios and when there 
is a potentially substantial time delay between command 35 
generation and task execution as occurs for example in 
surface controlled undersea and ground controlled 
space robotic operations. 

One convention approach would be to develop an 
interpretive robot language which can then be used to 40 
write a program which will execute on the robot to 
execute a specific task. The program would be sent to 
the system executive which would execute the program 
to control the robot. The interpretive language ap- 
proach may be desirable for use in factory automation 45 
and similar tasks where a specialized algorithm may be 
needed for each task. 

One major improvement in this area has been the 
development of systems, as described for example in 
U.S. patent application Ser. No. 07/699,299, filed May 50 
9, 1991 by the inventor hereof, in which series of task 
primitive parameters are transmitted between control 
and operation locations to provide task execution con- 
trol by one or more task execution primitives and obvi- 
ate the need to prepare and transmit new robotic con- 55 
trol programs for each new task. 

The task execution primitive approach is particularly 
suited for space telerobotics applications where flight 
qualifying software is required. In this case, the soft- 
ware for the task execution system which resides in 60 
space is fixed and can be completely flight qualified 
before the missions. Specific applications are then speci- 
fied by the new set of task execution parameters which 
are transmitted to the space vehicle to describe the 
desired robot behavior without the need for new pro- 65 
gramming. 

A task execution primitive may be described as a 
function which controls a manipulator to perform the 
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task described by its input parameter set. The primitive 
generates the desired setpoints and performs the desired 
control. The parameter list is the interface between a 
higher level task planning system and task execution. 
The details of the implementation are hidden from the 
planning system. The planning system only needs to 
know how to describe the desired behavior of execution 
by setting the input parameters of the task primitive. 
The command to execute the task primitive and the 
input parameter list are received from the planning 
system by a system execute which starts execution of 
the primitive and returns periodic execution status re- 
ports. 

In general, remote control of such robotic operation 
may be accomplished by teleoperation, autonomous or 
supervisory control as well as a combination of these 
approaches which is known as shared control. 

Interactive robotic task planning, execution and mon- - 
itoring can be accomplished with pure teleoperation. In 
this approach, planning resides within the operator’s 
mind, execution is issued by the operator via hand con- 
trollers and monitoring is provided by sensory feedback 
to the operator. Autonomous task planning, execution, 
and monitoring is the other extreme to teleoperation. 
Here, the operator initiates only very high level com- 
mands such as “replace the electronics module” and 
planning, execution, and monitoring is then done auton- 
omously without further operator input. 

Teleoperation has proven to be a valuable tool for 
many tasks especially in unmodeled or poorly modeled 
environments and for unplanned scenarios. The increas- 
ing complexity of the tasks to be performed places an 
ever increasing burden on the operator. Autonomous 
control is becoming increasingly valuable as a tool to 
relieve the operator of many task planning, execution, 
and monitoring responsibilities in order to allow the 
operator to concentrate on the more crucial elements of 
a task. 

Supervisory and shared control are recent improve- 
ments in telerobot task execution for unplanned scenar- 
ios, or for poorly modeled environments. Supervisory 
control is where the operator selects autonomous con- 
trol commands and associated parameterization for a 
task and can stop execution at any time. Shared control 
is the mixing of inputs from an operator and an autono- 
mous control system during task execution. 

A key element needed for planning, execution, and 
monitoring in a supervisory or shared control system is 
an operator interface which supports shared and super- 
visory control features. Supervisory features are re- 
quired to permit the operator to set up teleoperation, 
autonomous, and shared control task environment pa- 
rameters and to provide specific input parameters for 
autonomous task primitives and teleoperation control. 

Supervisory and shared control systems benefit 
greatly from the use of task primitives, which are reus- 
able, predetermined, self contained, preprogrammed 
programs for controlling a robot to accomplish various 
tasks, such control being dependent upon a set of input 
parameters which may be designated before, at the 
beginning time or during task execution. Shared fea- 
tures of an operator interface are required in order to 
provide autonomous setting of some environment and 
control parameters depending upon task context. 

The utility of a particular task primitive depends 
upon it’s flexibility. The utilization of sensors, both real 
and virtual, enhances task execution capability both by 
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providing alternate approaches for executing the task 
and by making task execution more robust. 

A very simple robotic system might have purely 
position control of a robot from a trajectory generator. 
Adding a hand controller allows the operator to per- 5 
form position teleoperation. A force-torque sensor 
makes force/compliance control possible and therefore 
robust contact tasks. A virtual force field sensor can aid 
the operator during teleoperation to keep the robot 
away from joint limits and objects. Individual task prim- 10 
itives may be provided in the remote robot system for 
each required task, but the limited size of such remote 
robot systems, in terms of computer memory limitations 
and/or number of line of programming code that may 
be incorporated therein, limits the number of such spe- 15 
cialized task primitives which may be used. 

It is often difficult to include multiple sensors in a 
robot control system due to the added system complex- 
ity, the difficulty in programming the robot to utilize 
the sensor, and the difficulty in specifying the task to 20 
utilize the sensor. What are needed are task execution 
primitives which simplify task description and execu- 
tion when utilizing multiple sensory inputs in addition 
to trajectory information, especially task execution 
primitives for the planning, execution and monitoring of 25 
telerobot tasks in poorly modeled environments and for 
unplanned scenarios. Such task execution primitives 
should efficiently and conveniently permit the combina- 
tion of teleoperation, autonomous, supervisory, and 
shared control techniques. 30 

BRIEF STATEMENT OF THE INVENTION 

The preceding and other shortcomings of the prior 
art are addressed and overcome by the present inven- 
tion that provides, in a first aspect, a method of operat- 35 
ing a telerobot by providing a set of input parameters 
from a local control site to a remote execution site in- 
cluding a telerobot, providing a general motion primi- 
tive at the remote site for controlling the telerobot in 
response to the set of input parameters and to remote 40 
and local sensor data, generating trajectory motion 
input at the remote site with the general motion primi- 
tive in response to input parameters specifying desired 
telerobot trajectory behavior, generating remote sensor 
motion input at the remote site with the general motion 45 
primitive in response to a combination of input parame- 
ters specifying desired telerobot behavior based on re- 
mote site sensor information and sensor data originating 
at the remote site, generating local sensor motion input 
at the remote site with the general motion primitive in 50 
response to a combination of input parameters specify- 
ing desired telerobot behavior based on local site sensor 
information and sensor data originating at the local site 
and simultaneously combining the trajectory, remote 
sensor and local sensor motion inputs at the remote site 55 
to control the motion of the telerobot. 

The generalized motion primitive of the present in- 
vention provides a maximum of parameterized motion 
capabilities with a minimum size remote robot system. 
The input parameter set specifies the desired behavior 60 
of each motion module or subsystem as well as the 
desired monitoring. The generalized Motion primitive 
replaces the multiple task specific primitives which 
each have a more limited capability. 

The generalized motion primitive eliminates the re- 65 
dundancy inherent in the set of task specific primitives 
it replaces and provides capabilities not available in the 
set of task specific primitives. The reason for the greater 
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capability is that the general motion primitive provides 
the capabilities of the individual primitives it replaces as 
well as permutations of those abilities resulting from the 
combination of the non-redundant portions of the com- 
plete set of such task specific primitives. 

These and other features and advantages of this in- 
vention will become further apparent from the detailed 
description that follows which is accompanied by a set 
of drawing figure(s). In the figures and description, 
numerals indicate the various features of the invention, 
like numerals referring to like features throughout both 
the drawings and the description. 

BRIEF DESCRIPTION OF THE DRAWING(S) 

FIG. 1 is a block diagram schematic of a telerobot 
control system utilizing a task execution primitive ac- 
cording to the present invention. 

FIG. 2 is an illustration of an example of a task to be 
performed in accordance with a general motion primi- 
tive according to the present invention. 

FIG. 3 is a block diagram outline of a general motion 
primitive in accordance with the present invention. 

DETAILED DESCRIPTION OF THE 
INVENTION 

A complete telerobot control system utilizing task 
execution primitives will first be described and then the 
details of the specific task execution primitive of the 
present invention will be described in greater detail. 

Referring now to FIG. 1, telerobot control system 10 
includes operator interface 12 and remote task execu- 
tion system 14 which includes robot manipulator 16, 
suitable for the tasks to be performed, and communica- 
tion link 18 for providing bidirectional communication 
with operator interface 12. Robot manipulator 16 in- 
cludes one or more remotely controllable devices such 
as robot arm 20, end effector 21 and remote sensors, 
such as force-torque sensor 19 and video camera 22. 
Communication link 18 includes communication and 
control system processor 24 as well as a suitable linkage 
device, such as antenna 26. 

Operator interface 12 includes one or more separate 
or combined operator terminals, including setup, simu- 
lation and execution terminals 28, 30, and 32. Operator 
interface 12 also includes operator interface communi- 
cation processor 34, including a suitable linkage device 
such as antenna 36, as well as graphics simulator 38 and 
local robot manipulator 40 which are described in 
greater detail in copending U.S. patent application Ser. 
No. 07/699,299, filed May 9, 1991, referenced above. 

Each operator terminal, such as execution terminal 
32, includes monitor central processor 44, a keyboard, 
mouse or other data entry device such as terminal entry 
device 46, and a hand control and feedback device, such 
as hand controller 48. During the various operations 
performed at the operator terminals, the operator enters 
information into the appropriate terminal via terminal 
entry device 46 and/or hand controller 48 and receives 
information from remote task execution system 14, 
graphics simulator 38, and/or local robot manipulator 
40 via monitor 42 and/or hand controller 48, as appro- 
priate to the task. 

All the devices in operator interface are intercon- 
nected by a conventional interconnection system, desig- 
nated generally as interconnection system 50. 

Telerobot control system 10 is operable in a conven- 
tional telerobot control mode in which pure teleopera- 
tion control is employed. During such operation, an 
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operator working at execution terminal 32 is directly 
connected in real time, or near real time, to remote task 
execution system 14 via operator interface communica- 
tion processor 34. Commands entered via terminal entry 
device 46 and/or hand controller 48 are implemented 5 
by robot manipulator 16 while information is made 
available to the operator from remote sensors, such as 
force-torque sensor 19 and/or video camera 22, via 
monitor 42 and/or hand controller 48. 

Telerobot control system 10 is operable in a conven- 10 
tional autonomous mode in which autonomous control 
is employed. During such control, the operator applies 
an input to execution terminal 32 to cause the execution 
of a preprogrammed task by robot manipulator 16. The 
input is applied by interconnection system 50 and opera- 15 
tor interface communication processor 34 to communi- 
cation link 18 which initiates the execution of the pre- 
programmed task within communication and control 
system processor 24 by robot manipulator 16. Some 
information relating to the status of the task and robot 20 
manipulator 16 may be collected by sensors, such as 
video camera 22, and communication and control sys- 
tem processor 24 and be provided to the operator by 
monitor 42. 

Telerobot control system 10 is operable in a conven- 25 
tional supervisory mode in which autonomous task 
execution is initiated by the operator by the provision of 
the appropriate parameters to communication and con- 
trol system processor 24 via terminal entry device 46 
and/or hand controller 48. The preprogrammed task 30 
within communication and control system processor 24 
initiated in this mode is in the form of a task execution 
primitive, in accordance with the present invention, 
which requires the provision of a set of input parame- 
ters. These parameters may be modified during execu- 35 
tion. 

Remote task execution system 14, in addition to a set 
of task primitives, includes an executive program simi- 
lar to executives within graphics simulator 38 and local 
robot manipulator 40. These executives control one or 40 
more central processing units, or CPU’s, within remote 
task execution system 14 and/or operator interface 12, 
which sequentially call, operate and execute task primi- 
tives, provide status and sensor feedback to operator 
interface 12 and run other programs, in sequence or in 45 
parallel with the task primitives, such as embedded 
safety and monitoring programs. The executive also 
maintains and uses a database of global parameters 
which are normally changed less frequently than the 
parameters provided to a task primitive would be 50 
changed. 

The use of a set of task primitives common to opera- 
tor interface 12 and remote task execution system 14 
permits the interactive development and control of the 
operation of tasks by robot manipulator 16 by passing 55 
only parameters and teleoperation inputs between oper- 
ator interface 12 and remote task execution system 14. 
Control of remote task execution system 14 by parame- 
terization which may be developed and tested at a local 
site before being applied to remote task execution sys- 60 
tern 14 permits a relative wide range of operation of 
robot manipulator 16 without the transmission of new 
programming from the local to the remote site. The 
sequence of parameterized task primitives may be se- 
quenced or stepped through using the executive in oper- 65 
ator interface 12 or in remote task execution system 14. 

Telerobot control system 10 is operable in a shared 
control mode in which teleoperated inputs are merged 
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during task execution with inputs from autonomous task 
primitives. Execution of shared control is initiated by 
the operator by provision of the appropriate parameters 
to communication and control system processor 24 via 
terminal entry device 46 and/or hand controller 48. The 
preprogrammed task primitive within communication 
and control system processor 24 is then provided with 
additional inputs from the operator during execution of 
the task. 

Task primitives under supervisory and/or shared 
control may be selected in a predetermined sequence to 
create more complex tasks. For example, a guarded 
motion primitive may conveniently be followed by a 
compliant motion primitive, according to the present 
invention, controlling the grasp of end effector 21 
mounted on robot arm 20 to create a more complex 
task. 

A compliant motion primitive for controlling the 
grasp of end effector 21 could then be used to adjust the 
position of end effector 21 to control the contact forces 
while closing the end effector gripper fingers. Sensory 
feedback provides information concerning the contact 
forces and torques while parameterization of the com- 
pliant motion task primitive sets the control parameters. 

Supervisory control to move robot manipulator 16 
requires selection of individual task primitives, their 
sequencing, and parameterization of each such task. 
Shared control while executing a group of tasks permits 
the addition of further teleoperation input by the opera- 
tor during execution of the task. 

Shared control with task primitive sequencing per- 
mits the execution of complex tasks in response to un- 
planned scenarios and/or in poorly modeled environ- 
ments, based on combinations of teleoperation with a 
group of preprogrammed, generalized task primitives, 
such as guarded and compliant motion. A generalized 
compliant motion with sensory fusion primitive in ac- 
cordance with the present invention is described below 
in greater detail with regard to FIG. 2 and FIG. 3. 

Development of a complex task from the sequencing 
of a series of less complex task primitives may be ac- 
complished in telerobot control system 10 interactively, 
using either teleoperation and/or editing, to provide 
appropriate parameterization. Feedback of the opera- 
tion of local robot manipulator 40 in response to the 
developed task may be provided by the appropriate 
sensors, similar to force-torque sensor 19 and associated 
with local robot manipulator 40 and/or setup terminal 
28. 

A substantial benefit of the flexibility of further devel- 
opment provided by telerobot control system 10 is that 
a relatively fixed set of preprogrammed task primitives, 
fully tested and qualified for use with remote task exe- 
cution system 14, can later be utilized as necessary by 
altering the parameterization of the task primitives to 
meet the robot control requirements of a new poorly 
modeled environment and/or unplanned scenarios. The 
developments of such new instantiations of the prepro- 
grammed task primitives, that is, task primitives with 
new parameterizations, can be performed without risk 
or unreasonable burden of prior training on the part of 
the operator. 

Remote task execution system 14 includes communi- 
cation and control system processor 24 to control robot 
manipulator 16 and provide a family of task execution 
primitives as well as an executive program resident 
therein. In space operations, for example, it is very 
important that the set of primitives in the remote loca- 
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tion, such as a space vehicle, have been fully tested and 
are capable of being combined to perform all the tasks, 
planned or unplanned. 

While the particular family of task primitives may 
vary depending upon the application intended for each 5 
particular system, a family of such primitives will be 
described generally in order to facilitate an understand- 
ing of the present invention. In addition to the family of 
task primitives and executive task sequencing, remote 
task execution system 14 provides periodic task status, 10 
sensor data and other feedback information as well as 
command result information to operator interface 12 via 
communication link 18 and operator interface commu- 
nication processor 34. 

As described above, the task primitives may include 15 
guarded and compliant motion primitives. In accor- 
dance with the present invention, a generalized-compli- 
ant-motion or GCM primitive is provided for perform- 
ing compliant motion tasks in Cartesian space. The 
required inputs include the selection of the robot actua- 20 
tor, the coordinate frame to be used for the destination, 
the frame to be used for interpolated motion, the frame 
to be used for control, selection of time or velocity 
based motion, selection of time or velocity for posi- 
tional motion, position-force selection vector to select 25 
position and force degrees of freedom (DOF’s) in the 
control frame, compliance selection vector to select 
which position DOF’s also have compliance, force- 
compliance control gains, gains for virtual springs, 
force-torque and position-orientation thresholds, and 30 
ending conditions including a selection integer selecting 
which ending conditions to test for, such as maximum 
errors in position, orientation, force and torque and 
their rates of change. 

The compliant grasp primitive closes the gripper 35 
fingers of end effector 21 while performing force con- 
trol to control contact forces. Inputs include which 
robot actuator to be selected, gripper type such as pneu- 
matic or servoed, selection of frame in which to do 
force control, force control gains and force control 40 
setpoints, and force-torque and position-orientation 
thresholds. A similar primitive is the free grasp primi- 
tive which simply opens or closes the gripper portion of 
end effector 21. 

Telerobot control system 10 provides a hierarchical 45 
menu system to guide the operator during description 
or development of a task from general motion types at 
the top of the hierarchy to specific motion types at the 
bottom of the hierarchy. The result is the specification 
of the task primitives and their parameterization to 50 
perform the tasks desired by the operator. The operator 
need not know the specific task primitives to be used. 
Instead, the operator specifies a generic motion type, 
e.g. guarded motion, move to contact, compliant mo- 
tion, force reflecting teleoperation, or grasp. A new 
menu then is provided with interaction germane to the 
specific motion type. 

For example, if the operator specifies compliant mo- 
tion, the compliant motion menu may present hinge, 
slide, screw, insert, level, push, translation, and similar 60 
options. The operator’s selection of one of these options 
invokes a new menu with input selections pertaining 
only to that type of motion. The insert menu permits the 
operator to select the insertion direction, et cetera. The 
interactive hierarchical approach substantially reduces 65 
the number of decisions to be made by the operator at 
any one point in time while still permitting the develop- 
ment of a relatively complex, specific task. 
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Shared control is where teleoperation inputs from 
hand controller 48 are merged in communication and 
control system processor 24 during execution with the 
autonomous control provided by communication and 
control system processor 24 to control robot manipula- 
tor 16. At the same time, sensor data feedback from 
robot manipulator 16 can be provided to the operator 
via execution terminal 32 and/or hand controller 48. 

Telerobot control system 10 was configured for oper- 
ation with various time delays between teleoperation 
and command inputs and task execution as would be 
encountered in certain environments, such as those 
encountered in space and undersea projects. The actual 
time delays encountered are, of course, the result of the 
actual delays present between remote task execution 
system 14 and operator interface 12 which may change 
at unknown rates. To provide enhanced abilities to 
develop and simulate the operation of complex tasks in 
such environments, controllable variable delays, such as 
time delays 68 and 70, are provided in graphics simula- 
tor 38 and/or local robot manipulator 40, respectively. 

The generalized compliant motion with sensor fusion 
primitive, or GCMSF primitive, according to the pres- 
ent invention, will next be described with reference to 
FIG. 2 and FIG. 3. 

The GCMSF primitive of the present invention has a 
rich input parameter set to provide for the execution of 
a wide variety of specific tasks and conditions. The rich 
environment of the GCMSF primitive provides for the 
execution of tasks such as door opening, crank turning, 
bolt seating and turning, pushing, sliding, pin insertion 
and removal, and leveling. 

The GCMSF primitive of the present invention pro- 
vides six sources of robot motion which can be used 
individually or simultaneously. These sources of motion 
have two basic types: nominal motion trajectory gener- 
ation and sensor based motion. Trajectory generation 
provides the setpoints for the motion. Sensor based 
motion perturbs the nominal motion based on sensor 
feedback. Additionally, monitoring checks that the 
execution is proceeding safely and stops the motion if an 
anomalous condition is detected. 

Two motion time segments are used. The nominal 
motion time segment includes the position trajectory 
generation. After the nominal motion is complete, the 
ending motion time segment begins and continues for a 
specified time or until specified ending conditions are 
satisfied. 

The input parameters include system, trajectory, fu- 
sion, sensor, and monitor parameter types. System pa- 
rameters describe the complete primitive, trajectory 
parameters describe the desired setpoint generation, 
fusion parameters describe how to fuse the various mo- 
tion sources. Sensor parameters describe sensor data 
55 analysis and control and monitor parameters describe 
how to monitor task execution. 

The present invention uses a split rate force-torque 
control technique using force and torque data read from 
a 6-axis force-torque sensor. Gravity compensation 
using input load mass properties is used to determine 
contact forces. Cartesian space motion and force con- 
trol is used. 

Referring now in particular to FIG. 2, an example of 
end effector 21 will be described together with the 
object to be manipulated in an exemplary task in order 
to specify various coordinate frames used in the descrip- 
tion of the GCMSF primitive. The task to be used in 
this example is the task of rotating crank 72 about axis 
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74. Crank 72 includes knob 76 suitable for grasping for 
rotation by the terminal link, such as grip tool 78, of 
robot arm 80. 

Robot arm 80 is illustrated in a functional form in 
FIG. 2 and includes wrist joints 82, links 84 and 86, and 5 
fixed first link or base 88 all supported by support 90 for 
positioning a terminal link such as grip tool 78. The 
primary Cartesian coordinate frame is shown as the 
WORLD coordinate frame. The coordinate frame of 
the point of connection between link 86 and base 88 is 10 
the BASE coordinate frame, while the coordinate 
frame of wrist joint 82, the terminal link of arm 80, is 
shown as the TN coordinate frame. The coordinate 
frame of the work piece, that is, crank 72, is shown as 
the NOM coordinate frame. Axis 74 is along the vertical 15 
axis of the NOM coordinate frame. 

The following parameters are included in the rich 
input parameter set. 

System parameters include trBase which provides the 
transformation from the WORLD coordinate frame to 20 
the BASE coordinate frame of robot arm 80; massProp 
which provides the mass properties of the task depen- 
dent load beyond force-torque sensor el; and period 
which indicates the desired reporting period for status 
reports to be provided to the executive program. 25 

There are five trajectory parameters. The trajectory 
parameter trTnDest is the transform from the WORLD 
coordinate frame to the destination TN coordinate 
frame. The trajectory parameter trNom is the transform 
from the TN coordinate frame to the NOM coordinate 30 
frame. The trajectory parameter timeVelSel selects 
time or velocity based motion. The trajectory parame- 
ter timeVelVal is the value of the time or velocity to 
execute nominal motion and the trajectory parameter 
accTime is the time to ramp up to maximum velocity. 35 

Input parameters related to sensor fusion include the 
sensor fusion control parameter maxSfVel which is the 
maximum velocity in the NOM coordinate frame due to 
the sum of all sensor based control and the sensor fusion 
monitor parameters transThreshold, which is the maxi- 40 
mum translation motion of the NOM coordinate frame 
due to sensor input, and angThreshold which is the 
maximum angular motion of the NOM coordinate 
frame due to sensor input. 

There are many termination condition parameters in 45 
the input parameter set. The parameter select provides 
bit mask selecting for the termination conditions to be 
tested for. testTime sets the time over which to average 
termination conditions. endTime is the maximum time 
to test termination conditions after nominal motion. 50 

endTransErr is the minimum translation error in the 
NOM coordinate frame. endAngErr is the minimum 
orientation error in the NOM coordinate frame, end- 
TransVel is the minimum time derivative of endTran- 
sErr while endAngVel is the minimum time derivative 55 
of endAngErr. 

endForceErr is the minimum SENSE frame force 
error vector magnitude and endTorqueErr is the mini- 
mum SENSE frame torque error vector magnitude. 
endForceVel is the minimum time derivative of end- 60 
ForceErr while endTorqueVel is the minimum time 
derivative of eadTorqueErr. 

The force sensor control input parameter set includes 
trForce which is the transform from NOM coordinate 
frame to the FORCE frame where force control is 65 
applied and trSense which is the transform from NOM 
coordinate frame to the SENSE frame where sensed 
contact forces are transformed. selVectFc is the selec- 
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tion vector selecting position or force control for each 
DOF of the FORCE frame. complyVect selects which 
position controlled DOFs of the FORCE frame are to 
have compliance, i.e., additional force control. 

forceSetpoints provide the force and torque setpoints 
in the FORCE frame while forceGains provides the 
force control gains in the FORCE frame. The force 
sensor control parameter deadZone provides the dead 
zone filter forces and torques. maxForceVel provides 
the maximum velocities in the FORCE frame DOFs 
due to force control. 

Force sensor monitor parameters include minFor- 
ceThres which is the minimum force vector magnitude 
in the FORCE frame, minTorqueThres which is the 
minimum torque vector magnitude in the FORCE 
frame and maxForceThres and maxT orqueThres which 
are the maximum force and torque vector magnitudes in 
the FORCE frame, respectively. 

Dither sensor control parameters include ditherWave 
which selects the shape of the dither waveform, 
trDither is the transform from the NOM coordinate 
frame to the DITHER frame for the input dither signal. 
ditherMag is the dither wave magnitude in each DOF 
of the DITHER coordinate frame while ditherPeriod is 
the dither wave period in each DOF of the DITHER 
coordinate frame. 

The virtual springs sensor control parameter set in- 
cludes selVectSp which indicates the DOFs of the 
NOM frame in which to apply springs, springGains for 
the position and orientation spring gains to pull the 
NOM coordinate frame back to the nominal motion 
trajectory and maxSpring Vel which are the maximum 
velocities due to virtual springs. 

There are many input parameters related to teleoper- 
ation sensor control. teleMode sets the mode of shared 
control which may for example be the tool, world or 
camera mode. trCamera is the transform from the 
WORLD coordinate frame to the CAMERA coordi- 
nate frame. trTeleop is the transform from the NOM 
coordinate frame to the frame to apply teleoperation 
inputs. selVectTP is the selection vector to specify 
DOFs of TELEOP frame to add teleoperation inputs. 
teleGains is the parameter setting the gains for teleoper- 
ation inputs. maxTeleVel sets the maximum velocities 
due to teleoperation inputs in the TELEOP frame. 

Joint sensor control parameters include selVectLim 
and SelVectSing, the selection vectors selecting which 
joint limits and singularities to bounce away from. In 
addition, jsGain provides the force field gain and 
jsTheta is the distance from joint threshold to initiate 
joint limiting. 

A joint sensor monitor input parameter is provided in 
the form of jSafetyLimit which sets the safety margins 
from singularities and joint limits. 

Referring now to the block diagram of a preferred 
embodiment the GCMSF primitive shown in FIG. 3, 
the overall architecture of the GCMSF primitive will 
first be described. The control of the various inputs will 
then be described in greater detail. 

The GCMSF primitive provides six sources of robot 
motion which can all be used individually or simulta- 
neously. These sources of motion have two basic types: 
nominal motion trajectory generator and sensor based 
motion. 

Positional trajectory generator 102 provides a feed- 
forward Cartesian nominal position X<* of the NOM 
coordinate frame. This position is combined with data 
from the various sensors and subsystems, to be de- 
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scribed below in greater detail, in summer 104 and then 
converted in inverse kinematics generator 106 to the 
appropriate joint angle inputs Q c which are applied to 
robot manipulator 108. Robot manipulator 108 includes 
robot arm 80, as shown in FIG. 2, and robot control 5 
system 110 which applies control signals to arm 80 and 
receives joint feedback © a therefrom. 

The particular sensors shown in FIG. 3 include dither 
function generator 112, teleoperation input 114, force/- 
compliance control subsystem 116, joint limit control io 
118, and restoration springs subsystem 120. The output 
data from these sensors are combined in sensor summer 
122 and then applied to summer 104 through integrator 
105. In this manner, each of the sensors provides a per- 
turbation to the nominal position X</in the NOM coor- 15 
dinate frame. These perturbations are combined at the 
current position in the NOM coordinate frame and 
integrated with past cumulative sensor based motion. 
Restoration springs subsystem 120 tries to reduce the 
integrated cumulative sensor based motion. 20 

Sensor summer 122 uses input parameters to deter- 
mine how to merge inputs from the various motion 
sources. Sensor summer 122 may be expanded to per- 
form more complex merging of the inputs dependent on 
system state information or information from other 25 
sources either at the local or remote site. 

The motion is programmed using the following kine- 
matic ring equation in accordance with techniques well 
known in this art. 

30 

trBase-trTn 'trNom’trDeltrDrive — trBase>trTnDest-tr - 
Nom (1) 

As shown for example in FIG. 2, the WORLD coor- 
dinate frame is a fixed coordinate frame. trBase is the 
constant transform from WORLD coordinate frame to 35 
the BASE frame fixed in the fixed first link, base 88, of 
robot manipulator 108. trTn is the variable transform 
from BASE to the TN frame fixed with respect to grip 
tool 78, the terminal link of robot manipulator 108. This 
transform changes each sample interval during control 40 
and is computed based on the results of all other inputs. 

trNom is the constant transform from the TN frame 
to the frame in which Cartesian interpolated motion 
will occur, that is, the NOM coordinate frame. trDel is 
the variable transform which includes the integration of 45 
all sensor motion. trDrive is the variable transform 
which provides Cartesian interpolated motion. This 
transform is initially computed to satisfy the initial con- 
ditions of the transforms in the ring equation, equation 
(1) above, and is interpolated to the identity transform 50 
at the end of the nominal motion. 

trDest is the constant transform used to specify the 
nominal destination of the TN frame, that is, the ex- 
pected value of the trTn at the end of the nominal mo- 
tion. 55 

Referring now again to FIG. 3, at each sample inter- 
val, positional trajectory generator 102 calculates 
trDrive. The outputs of dither function generator 112, 
teleoperation input 114, force/compliance control Sub- 
system 116, joint limit control 118, and restoration 60 
springs subsystem 120 are combined in sensor summer 
122 to determine trDel from sensor based motion. trTn 
may then be computed by solving the ring equation, 
equation (1) above. trTn is applied to inverse kinematics 
generator 106 to derive 0 C which is then applied to 65 
robot control system 110 to drive robot arm 80. 

Most of these sources of input can specify these in- 
puts in a coordinate frame specific to their functionality; 
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nominal motion is specified in NOM, teleoperation in 
TELEOP, force-torque control in FORCE, etc. This is 
useful because these inputs may be most effectively 
specified in separate frames. 

In the example shown in FIG. 2, nominal motion is 
the motion of about axis 74 of, for example, crank 72. 
Linear interpolation of a fixed rotation will therefore 
cause motion in an arc. Force control is provided at 
knob 76 where grip tool 78 at the terminal link of robot 
arm 80 grasp crank 72. If, instead, the NOM coordinate 
frame were also at knob 76, interpolated motion would 
be in a straight line to the destination, rather than in an 
arc as it will actually be during the physical turning of 
crank 72 about axis 74. 

In addition, during the teleoperation mode of control, 
different situations may require that the operator con- 
trol the motion at knob 76 or at axis 74. If the input 
motion from the hand controller, such as hand control- 
ler 48 shown in FIG. 1, were mapped to motion about 
axis 74 then only one DOF input will be required from 
the operator. 

There are two major time segments of motion during 
the execution of the GCMSF primitive: the nominal 
motion segment and the ending motion segment. When 
the GCMSF primitive is initiated, it first executes the 
nominal motion segment with the specified Cartesian 
interpolated motion and inputs from sensors 112, 114, 
116, 118, and 120. 

Motion is halted if a monitor event is triggered or 
when a prescribed Cartesian interpolated motion is 
completed. If this nominal motion segment completes 
normally, then the ending motion segment begins. Ex- 
actly the same control occurs except that there is no 
Cartesian interpolated motion because only the sensor 
based motion is active. 

In other words, during the first or nominal motion 
time segment, positional trajectory generator 102 and 
sensor summer 122 provide motion control input infor- 
mation to summer 104. Upon successful completion of 
the Cartesian motion requested by positional trajectory 
generator 102, the final or ending motion time segment 
begins in which the only active input to summer 104 is 
from sensor summer 122. 

During the nominal motion segment, termination 
conditions are not tested. During the ending motion 
time segment, termination conditions are tested so that 
motion can be stopped as a result of a monitor event, 
expiration of a prescribed time period, or the occur- 
rence of a termination condition. 

The ending motion time segment is required in order 
to have a task finish with a known state, known to be 
satisfying or not satisfying the specified acceptable end- 
ing conditions. Further, testing for termination condi- 
tions may not be desired until the nominal motion seg- 
ment is complete. 

Control of robot arm 80 may be accomplished with 
multiple rates of control. For example, the Cartesian 
level control which includes all control associated with 
the GCMSF primitive may be run with a 5 ms sample 
interval while the joint servo control in robot control 
system 110 may have a different sample interval, such as 
a 1 ms sample interval. 

Now that the general architecture for control in the 
GCMSF primitive has been described, the control in 
each individual sensor will be described in greater de- 
tail. 
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Motion control will be discussed first. Positional tra- 
jectory generator 102 uses the known RCCL trajectory 
generator described in detail in a paper by Lloyd et al. 
entitled “Extending the reel programming environment 
to multiple robots and processors” published in Proa 5 
IEEE Inf l Conf. on Robotics and Automation pp 
465-474, 1988. Positional trajectory generator 102 gen- 
erated the trDrive transform which is initially given by 
the following equation: 

10 

trDrive= ( trTnlnit-trNom ) “ l -trTnDest-trNom ( 2 ) 

where trTnlait is the initial value of trTn. trDrive is 
then linearly interpolated from this initial value to the 
identity transform at the end of the motion. This inter- 15 
polation is controlled by the input parameters timeVel- 
Sel, timeVelVal and accTime. timeVelSel selects 
whether to finish the motion in a specified time or with 
a specified velocity. timeVelVal is the time or velocity 
value to execute the motion in. accTime is the time to 20 
ramp up to maximum velocity. If desired, an additional 
input parameter may be specified to control the maxi- 
mum acceleration. 

Force control, and compliance control, are imple- 
mented in the same way and will now be described 25 
together. Force control is implemented independently 
in each DOF of the Cartesian force control frame 
FORCE. Compliance control is used to describe the use 
of force-torque control with zero force setpoints in 
DOFs which are being controlled by another input 30 
device, such as positional trajectory generator 102 or a 
teleoperation input device such as hand controller 48 
shown in FIG. 2. 

Force control is used to modify the position setpoint 
in each DOF in order to control the contact forces. This 35 
results in damping control with input force error and 
output position perturbation per sample. The result of 
force-torque control in each sample interval is the per- 
turbation transform trDelFc. 

The first step in achieving force-torque control dur- 40 
ing a sample interval is the projection of forces and 
torques, from the force-torque sensor frame to the 
SENSE frame. Force-torque sensor 81 is a 6 DOF wrist 
force-torque sensor associated with robot arm 80. 
Force-torque sensor 81 senses force and torque data in 45 
the SENSOR frame centered in force-torque sensor 81. 
Such force-torque data is then projected to equivalent 
forces in the TN frame using rigid body force transfor- 
mations. 

The forces on the load, that is, the complete compos- 50 
ite body beyond force-torque sensor 81 due to gravity 
are then computed. The mass and center of mass of the 
load with respect to the TN frame are given in the 
massProp input parameter. The current TN frame ori- 
entation with respect to the gravity vector is used with 55 
the load mass properties to determine the gravity load 
forces in the TN frame. The resultant forces and torques 
are those due only to contact and are force-torque sen- 
sor output f fl . 

Force-torque sensor output f a is in the TN frame and 60 
is applied to TN to SENSE transform subsystem 124 to 
provide sensor output f a with respect to the SENSE 
frame. The forces in the SENSE frame are applied 
through deadzone filter 126 which reduces their magni- 
tude by the values in the input parameter deadZone. If 65 
one of the force-torque magnitudes is initially less than 
the corresponding value in the input parameter dead- 
Zone, that value is set to zero. Deadzone filter 126 is 
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useful to reduce drift due to inaccuracies in the determi- 
nation of the mass properties of the load. 

Force control is calculated in the FORCE frame 
using the forces projected into the SENSE frame. The 
FORCE and SENSE frames will usually coincide, but 
there are cases where they may be different, such as the 
task of leveling a plate on a surface where the SENSE 
frame is at the center of the plate and the FORCE frame 
is at the point of contact. In that case, if the SENSE and 
FORCE frames were both at the point of contact, then 
no moments would be felt and therefore no rotation due 
to force-torque control would occur since the force line 
of action would be through the control frame. 

Force setpoint generator 128 provides force setpoints 
in each DOF of the FORCE frame in accordance with 
the forceSetpoints input parameter. The selVectFc 
AND comply Vect selection vector input parameters 
select which of the 6 DOFs of the FORCE frame are to 
have force and/or compliance control applied. In the 
selected DOFs, contact forces from deadzone filter 126 
are subtracted from the setpoints provided by force 
setpoint generator 128 in subtractor 130. 

The output of subtractor 130 are force errors which 
are multiplied in force feedback processor 132 by con- 
stants provided in the forceGains vector input parame- 
ter to produce a differential motion vector of six pertur- 
bations in the FORCE frame. The resultant three trans- 
lations and three rotations are given by the following 
equation: 

df=(dfc dfy, djx , SjOc, 6/s) (3) 

The magnitudes of the elements of the d/ vector are 
then limited in limiter 134. The maximum magnitudes of 
the d/perturbations per sample interval are the velocity 
limits given in the maxForceVel input parameter multi- 
plied by the sample interval. 

The output of limiter 134 is the FORCE/ r iKF C trans- 
form which is a differential translation and rotation 
transform with elements given by d/ as shown in the 
following equation: 

< 4 > 

1 6^ df x 

S/z 1 ~§fx dfy 

&fx ^ djz 

0 0 0 1 

The trDelFc transform is then applied to FORCE to 
NOM transform subsystem 136 and transformed to the 
NOM coordinate frame. trDelFc with respect to the 
FORCE and NOM frame are related by the following 
equation: 

N OM tr j) e ip c . tr p orce - s . tr p orce .FORCE tr jy e ip c ( 5 ) 

The trDel transform of equation (1) is then updated in 
sensor summer 122 with the perturbation due to force- 
torque control in accordance with the following equa- 
tion: 

trDel = NOM trDelFc-trDel ( 6 ) 

Premultiplication is required rather than postmulti- 
plication because the motion is with respect to the 
NOM coordinate frame. 


FOKCEtrDelFc = 
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Dither function generator 112 can be used to provide 
dither signals to perturb the motion independently in 
each DOF of the DITHER frame. Dither function 
generator 112 includes dither wave generator 138 
which provides the selected dither waveform, such as a 5 
triangular, sinusoidal or square wave, using the magni- 
tude and period of the dither waveforms for each DOF 
of the DITHER frame given in the ditherMag and 
ditherPeriod input parameters. As with force-torque 
control, the inputs in each DOF are elements of a differ- 10 
ential translation and rotation transform, trDelDt, 
which is transformed to the NOM frame by DITHER 
to NOM transform subsystem 140. The trDel transform 
is then updated with the perturbation due to the dither 
waveforms in sensor summer 122 in accordance with 15 
the following equation: 

trDel ~ N0M trDelDt-trDel (7) 

The operation of teleoperation input 114 will be de- 20 
scribed next. Teleoperation sensor 142 may conve- 
niently be a 6 DOF hand controller or two 3 DOF 
joysticks, such as hand controller 48 shown in FIG. 1. 

In each sample interval the change in joint angles of 
teleoperation sensor 142 are applied as a vector, A©/*, to 25 
Jacobian multiplier 144 for multiplication with the ap- 
propriate Jacobian in accordance with the teleMode 
input parameter to get the input Cartesian motion per- 
turbations, AX*. 

These motion perturbations are transformed to the 3 
TELEOP frame in TELEOP transform subsystem 146. 
The mode of operation, such as tool mode, or world 
mode, or camera mode, of teleoperation determines 
how the perturbations are to be transformed to the ^ 
TELEOP frame. The trCamera input parameter is used 
for camera mode teleoperation to specify the present 
operator viewing orientation. Additional information 
and details concerning modes of operation are provided 
in U.S. patent application Ser. No. 07/699,266, filed ^ 
May 9, 1991, of which this present application is a con- 
tinuation-in-part. 

The teleGains input parameter provides the 
weightings for the inputs. These weightings are applied 
in multiplier 148. The input parameter selVectTp selec- 45 
tion vector selects which DOFs of teleoperation inputs 
to include and the maxTelVel input parameter limits the 
velocity due to teleoperation inputs. These limits and 
selections are accomplished in limiter 150. 

The resultant teleoperation sensor input is trans- 50 
formed from the TELEOP to NOM frame in TELEOP 
to NOM transform subsystem 152 before application to 
sensor summer 122. The transform from TELEOP to 
NOM is given by the trTeleop input parameter. 

Joint sensor control is provided by joint limit control 55 
118 which prevents robot arm 80 from going into a joint 
limit or singularity. The input to joint limit control 118 
is © a which is applied by robot arm 80 as a feedback 
control signal to robot control system 110. 

Joint limits are provided by joint limit generator 154 60 
in the form of Ou m and combined with O a in subtractor 
156. The output of subtractor 156 is therefore an indica- 
tion of the approach of the joint angle to its limit and is 
equal to the difference between the actual joint angle, 

© c , and the joint limit value, ©// m , for each DOF. The 65 
output of subtractor 156 is applied as a vector quantity 
to multiplier 158 to determine the joint angle perturba- 
tion in accordance with the following equation: 


A0 = 


Kb 


( 8 ) 


where K# is the gain, Q a is the actual joint angle and 
©// m is the limit the joint is approaching as a joint limit 
or as a singularity. 

The differential vector output of multiplier 158 is 
multiplied by the appropriate Jacobian in Jacobian mul- 
tiplier 160 to get the required Cartesian motion. This 
Cartesian motion is transformed to the NOM coordi- 
nate frame in TN to NOM transform subsystem 162, 
appropriate limits are applied in limiter 164 and the 
result is added to trDel in sensor summer 122. 

With regard now to restoration springs subsystem 
120, virtual restoration springs act on the trDel trans- 
form to cause trDel to approach the identity transform. 
An identity transform is a transform such that when 
pre-or post-multiplied with a second transform, the 
result is equal to the second transform. 

This reduces the accumulated motion due to sensory 
inputs and causes the actual motion to approach the 
nominal motion. Virtual springs are applied in the 
DOFs specified in the selVectSp input parameter. In 
the preferred embodiment of the present invention 
shown in FIG. 3, four virtual springs are used, one 
along each translational DOF and one orientation 
spring. 

For the translational DOFs, the spring lengths are 
equal to the displacement vector p element of the trDel 
transform. trDel is a homogeneous transform with col- 
umn vectors n, o, zt, and p. The translation perturbations 
due to the virtual springs, d 5 , are then the spring lengths 
multiplied, in multiplier 166, by the translational spring 
gains in the springGains vector, k s , input parameter in 
accordance with the following equations: 

dsx~ —ksxPx 

dsy~ —ksypy 

d S z=—k sz p z (9) 

The virtual spring used for orientation is applied 
about one axis with respect to the NOM frame. The 
selection of this axis depends upon the number of orien- 
tation DOFs specified in the selVectSp input parameter. 
The axis is u and the angular displacement about this 
axis is 0. 

If all orientation DOFs are selected, then u is the 
equivalent axis of rotation of the trDel transform and 0 
is the equivalent angle about the axis. If no orientation 
DOFs are selected, then no orientation perturbation is 
applied due to virtual springs. If only one orientation 
DOF is selected, then the corresponding axis x, y, or z 
is aligned by the orientation virtual spring. 

The vector u and 0 are given by the following formu- 
las: 


axis 

u 

0 


X 

unit (ii X x) 

arccos (n • x) 


y 

unit (6 X y) 

arccos (6 * y) 


z 

unit (a X z) 

arccos (a * z) 

(10) 


where x= (1,0,0), y = (0,1,0), z= (0,0,1). The virtual 
springs orientation perturbation is then given by the 
following equality: 
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its magnitude is less than its associated input parameter 
dse=-k 2 e^ (li) limit. The endTransErr condition is the magnitude of 


where — kse is the orientation gain in the springGains 
vector input parameter. 

The four virtual springs perturbation magnitudes are 
then limited, in limiter 168, to the magnitudes given in 
the maxSpringsVel vector input parameter in as the 
force-torque control perturbations were limited by the 
MaxForceVel values. The output of limiter 168 is then 
applied to sensor summer 122 to update the trDel trans- 
form in accordance with the following equation: 

trDel ’= trans(x > d sx )-trans(y,d sy) • trans&d^rot- 
(u,dse>trDel ( 12 ) 

where trans(v,d) is a translation of d along the v axis and 
rot(v, 5) is a rotation of 8 about the v axis. 

Various parameters are continuously monitored dur- 
ing execution. The magnitudes of the translational part 
of trDel and the equivalent rotation of the orientation 
part of trDel are compared against the input parameters 
posThreshold and orientThreshold in fusion monitor 
subsystem 103. If the values grow larger than the 
thresholds, then the motion stops. 

Also, the vector magnitudes of the contact forces and 
torques in the SENSE frame are compared against max- 
ForceThres and maxTorqueThres in force monitor 123 
and motion stops if one of them is larger than the thresh- 
old. If the distance to a joint limit or singularity is less 
than the angles in the jSafetyLimit input vector as com- 
puted in joint monitor 79, then motion stops. 

Other separate monitors may be useful, such as a 
termination condition monitor, which may also be in- 
cluded in the monitors or subsystems shown. A termina- 
tion condition monitor is used during the end motion 
time segment. The termination condition monitor may 
utilize information from all sources of inputs, for exam- 
ple, information available to fusion monitor subsystem 
103, force monitor 123 and/or joint monitor 79. The 
end motion continues until all of the specified termina- 
tion conditions are satisfied or until the time limit given 
by the endTime input parameter has passed. 

The termination condition monitor is important both 
because it signals when to stop the motion upon attain- 
ing a state as specified by the input parameters, and 
because it specifies the reason motion was terminated. 
The cause of termination of the motion is important 
both as feedback to the local site and when the primitive 
is used as part of a motion sequence. When the primitive 
is used in one command in a sequence of commands 
from the local site, the decision to continue on to the 
next command of the sequence is dependent on whether 
the previous motion ended with an acceptable state. 

An alternate approach to having the various monitors 
described above is to have one monitor that checks for 
both safety and termination conditions. This single 
monitor would also specify the reason that motion was 
stopped. 

The select input parameter is a bit mask which selects 
which termination conditions to test for. Any combina- 
tion of termination conditions can be tested. All termi- 
nation conditions relate to forces and torques in the 
SENSE frame or sensor based motion specified by the 
trDel transform. 

Each termination condition is calculated as a moving 
average of data sampled each 200 ms over a window 
whose width is provided by the testTime input parame- 
ter. Satisfaction of a termination condition means that 


the trDel transform p vector including only the position 
DOF components. The endAngErr condition is the 
5 magnitude of the virtual restoration springs angular 
displacement, 0, described above. The endTransVel 
and endAngVel parameters axe the rate of change of the 
endTransErr and endAngErr conditions, respectively. 

The endForceErr and endTorqueErr parameters are 
1° the magnitudes of the force and torque error vectors in 
the SENSE frame including only the force controlled 
DOFs. The endForceVel and endTorqueVel parame- 
ters are the rate of change of the endForceErr and 
endTorqueErr conditions, respectively. 

15 The preferred embodiment of the generalized motion 
primitive of the present invention as described above 
provides fundamental motion modules or subsystems 
with inputs which describe the desired behavior of the 
telerobot for each fundamental motion module. The 
motion modules are arranged, for the purposes of this 
discussion, into three groups: a priori trajectory gener- 
ated motion, remote site sensor motion and local site 
sensor motion. 

Remote or local site sensor based motion may be 
5 caused by either real or virtual sensors. Real sensors are 
sensors generating information based on physical data 
while virtual sensors are sensors which generate infor- 
mation based on imaginary sensors. 

30 The motion modules included in the described em- 
bodiment include positional trajectory generation, te- 
leoperation, joint limiting, force/compliance control, 
restoration springs and dither. Additional motion mod- 
ules may be provided in accordance with the present 
35 invention and would fall into one of the above described 
three groups of modules. 

Each such additional motion module would be pro- 
vided with an input parameter set which would de- 
scribe its desired behavior and would provide motion 
4Q input to sensor summer 122 and/or summer 104. 

For example, a collision avoidance subsystem may be 
implemented similarly to the implementations shown 
for joint limit generator 154 or force setpoint generator 
128. A virtual sensor could be used to provide the dis- 
45 tance between objects and, based on those distance, 
input motions to sensor summer 122 could be generated 
to avoid collisions. 

The position trajectory generator also represents the 
option of generating trajectories with a hand controller 
50 at the local site and providing them as destinations at 
the remote site. A trajectory generator could then gen- 
erate a trajectory from the destinations as shown for 
trajectory generator. 

The force setpoints in force/compliance control sub- 
55 system 116 are generated as shown from the input pa- 
rameters but could vary with commands from the local 
site similar to how teleoperation generates varying set- 
point commands from the local site. 

Trajectory generator 102 is described as generating a 
60 position trajectory in Cartesian space but could also 
provide setpoints in another space such as joint space 
where desired robot joint angles would be generated. 
Similarly, trajectory generator 102 could use joint space 
rather than Cartesian space and may be merged with 
65 summer 104. 

An expanded monitoring capability may also be de- 
veloped in which each motion module may have a mon- 
itor associated with it or fewer monitors could be pro- 
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vided which monitor multiple system states. The moni- 
tors monitor execution status, system safety, and exter- 
nal event status. 

While this invention has been described with refer- 
ence to its presently preferred embodiment, its scope is 5 
not limited thereto. Rather, such scope is only limited 
insofar as defined by the following set of claims and 
includes all equivalents thereof. 

What is claimed is: 

1. A method of operating a telerobot, comprising the 10 
steps of: 

transferring a set of input parameters from a local 
control site to a remote execution site including a 
telerobot, said input parameters specifying desired 
telerobot trajectory behavior in Cartesian space 15 
including behavior based on remote and local site 
sensor information and desired termination condi- 
tions; 

retrieving a general motion primitive at the remote 
site for autonomous, remote site closed loop con- 20 
trol of the telerobot to perform a compliant motion 
task in response to the set of input parameters and 
to remote and local sensor data; 

autonomously generating trajectory motion input at 
the remote site with the general motion primitive in 25 
response to said input parameters specifying de- 
sired telerobot trajectory behavior, said trajectory 
motion input being held constant during an ending 
motion time segment at a value related to its final 
value during a nominal motion time segment oc- 30 
curring before the ending motion time segment; 

autonomously generating sensor specific remote sen- 
sor motion input at the remote site with the general 
motion primitive for each of a plurality of remote 
sensors with regard to a coordinate frame specific 35 
to that remote sensor in response to a combination 
of said input parameters related to each of said 
remote sensors specifying desired telerobot behav- 
ior based on remote site sensor information and 
sensor data originating at the remote site related to 40 
each of said remote sensors; 

autonomously generating sensor specific local sensor 
motion input at the remote site with the general 
motion primitive for each of a plurality of local 
sensors with regard to a coordinate frame specific 45 
to that local sensor in response to a combination of 
said input parameters related to each of said local 
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sensors specifying desired telerobot behavior based 
on local site sensor information and sensor data 
originating at the local site related to each of said 
local sensors; 

autonomously generating sensor specific virtual sen- 
sor motion input at the remote site with the general 
motion primitive for each of a plurality of virtual 
sensors with regard to a coordinate frame specific 
to that virtual sensor in response to a combination 
of said input parameters related to each of said 
virtual sensors specifying desired telerobot behav- 
ior based on virtual sensor information and virtual 
sensor data related to each of said virtual sensors; 
transforming each said sensor specific motion input 
into a common coordinate frame; 
merging all sensor specific motion inputs in said com- 
mon coordinate frame together to generate a sen- 
sor based motion input in said common coordinate 
frame; 

integrating said sensor based motion input with previ- 
ous sensor based motion input to form a cumulative 
sensor based motion input; 
resolving a kinematic ring equation to specify the 
Cartesian spatial relationships between the trajec- 
tory motion input and the cumulative sensor based 
motion input at the remote site to generate task 
level commands in Cartesian space for controlling 
the motion of the telerobot to perform said compli- 
ant motion task; 

transforming the task level commands at the remote 
site to produce joint angle commands to control 
the motion of the telerobot to perform the compli- 
ant motion task; 

generating nominal motion monitoring information at 
the remote site during the nominal motion time 
segment in response to input parameters, sensor 
data and the motion of the telerobot to determine if 
telerobot motion is within predetermined limits 
during the nominal motion time segment; and 
generating ending motion monitoring information at 
the remote site during the ending motion time seg- 
ment in response to sensor data and the motion of 
the telerobot to determine when to terminate 
telerobot motion in accordance with said input 
parameters specifying desired termination condi- 
tions. 
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